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FOREWORD 


The  Fort  Hood  Field  Unit  of  the  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  conducts  research  under  Army  Project 
2Q7f>  J74 JA775  and  provides  technical  advisory  service  for  the  Traininq 
and  Doctrine  Command  (TRADOC)  Combip^d  Arms  Test  Act^vit^  (TCATA)  .  This 
report  is  responsive  to  a  request  by  rhi^Ccsisnah^  systems  Test  Director¬ 
ate  oi  lv'AT«£>for  technical  assistance  in  the  conduct  of  the  automatic 
data  processing  (ADP)  traininq  evaluation  effort  for  the  Tactical  Opera¬ 
tions  System  (TOS)  Concept  Test,  TCATA  Test  FM  222.  The  U.S.  Army  cur¬ 
rently  has  under  develojsnent  a  number  of  automatic  data  processing 
a;?  systems  for  employment  in  tactical  missions.  One  of  these  is  the  TOS, 
which  is  designed  to  facilitate  the  Command  and  Control  decision  pro- 
ess  by  quickly  gathering,  analyzing,  and  disseminating  tactical  infor¬ 
mation  to  field  commanders  in  sufficient  time  to  allow  quick  reaction 
to  rapidly  developing  tactical  situations. 

The  purpose  of  the  TOS  Concept  Test  (FM  222)  was  to  provide  a  data 
base  to  evaluate  a  concept  of  -automated  data  processing  assistance  to 
the  division  staff.  A  limited  system,  called  TOS  Operable  Seqment 
(TOs£)  ,  was  used  as  the  tost  bed  for  conducting  the  test.  The  ADP 
training  program  was  to  provide  all  staff  user  personnel  assigned  to 
operate  the  TOS?  with  the  knowledge  and  skills  required  to  effectively 
perform  their  tasks  during  execution  of  the  test.  Findings  from  as¬ 
sessment  of  the  TOS 5,  staff  user  training  program  provided  information 
for  answering  training  evaluation  objectives  of  the  test. 
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TRAINING  EVALUATION  Of  THE  INDIVIDUAL  STAFF  USEh  ADR  TRAINING'.  PROGRAM 
IN  TCATA  TEST  FN  222  TACTICAL  OPERATIONS  SYSTEM  (TO.S)  CONCEPT  TEST 


BRIEF 


Requirement : 

To  obtain  data  that  permits  assessment  of  the  training  requirements 
necessary  for  the  division  to  employ  TOS2. 


Procedure : 

The  training  schedule  consisted  of  5  weeks  of  4  hours  per  day. 

(The  other  4  hours  of  each  workday  were  spent  in  routine  unit  duties.) 
Performance,  demographic,  and  opinion  data  were  collected  to  answ«  i  the 
requirement .  Performance  data  were  obtained  by  means  of  end-of-traimnq 
tests  of  trainees.  Demographic  and  opinion  data  were  obtained  from 
questionnaires  and  training  reports  made  out  by  the  trainees  and  evalu¬ 
ators  at  the  end  of  the  main  lessons,  at  the  end  of  the  training  program, 
and  at  the  end  of  the  test.  The  data  consisted  of  measures  of  of>erator 
per formance  in  processing  Intelligence  information  on  TOS*  input/output 
terminals,  and  classification  and  rating  scale  scores,  remarks,  opin¬ 
ions,  and  observations  concerning  the  training  program.  Descriptive  and 
inferential  statistical  tents  and  techniques  were  used  to  analyte  the 
data. 


Findings : 

The  effectiveness  of  the  training  program  was  degrade  by  insuffi¬ 
cient  total  hours  allocated  for  training,  the  use  of  4-hour  rather  than 
8-hour  daily  training  sessions,  the  compressed  training  schedule,  and 
the  dual  commitment  of  students  to  classroom  and  unit  duties.  The  ef¬ 
fectiveness  of  the  training  program  would  be  improved  if  the  5  weeks  of 
4  hours  per  day  were  changed  to  weeks  of  8  hours  per  day.  An  ex¬ 

panded  training  program  of  5  weeks  of  8  hours  per  day  should  be 
considered . 

The  quality  of  instruction  presented  during  training  was  rated  as 
excellent.  However,  because  the  ratio  of  students  to  input/output  de¬ 
vices  was  too  high  (3.3  to  1),  training  effectiveness  was  degraded. 


Trainees  rated  the  practical  exercise  method,  which  emphasised 
hands-on  training,  as  more  effective  than  the  classroom- lecture  method 
of  training.  The  proper  balance  between  classroom- lecture  training 
and  hands-on  training  must  be  determined  by  future  research. 

The  proficiency  level  achieved  by  enlisted  trainees  was  generally 
higher  among  those  who  possessed  one  or  more  of  the  following  charac¬ 
teristics:  previous  functional  background  experience:  the  appropriate 
primary  military  occupat ional  specialty  (MOS  968,  Intelligence  Analyst); 
and  higher  aptitude  scores  in  general  technical  (GT) ,  combat,  and  sur¬ 
veillance  and  communications  areas.  The  best  single  prediction  of  pro¬ 
ficiency  level  was  GT  score. 


Utilization  of  Findings: 

This  report  was  incorporated  as  part  of  TCATA  test  report  FM  222 
Tactical  Operations  System  (TOS)  Concept  Test  RCS  ATCD-8,  November  1977. 
Test  results  were  employed  as  a  part  of  the  data  from  which  to  develop 
requirements  and  recommendat ions  for  a  TOS  at  division  and  corps,  which 
were  presented  to  the  CY  77  Army  Systems  Acquisition  and  Review  Council 
II  (AS ARC  ID/Defense  Systems  Acquisition  and  Review  Council  II  (DSARC  II). 
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TRAINING  EVALUATION  OF  THE  INDIVIDUAL  STAFF  USER  ADP 
TRAINING  PROGRAM  IN  TCATA  TEST  FM  222  TACTICAL 
OPERATIONS  SYSTEM  (TOS)  CONCEPT  TEST 


INTRODUCTION 

Advances  in  military  technology  have  produced  the  means  for  con¬ 
ducting  tactical  operations  at  a  much  faster  pace  and  intensity  than 
was  formerly  possible.  This  acceleration  of  activity  means  that  the 
cocnand  and  control  system  mus  c  process  more  tactical  information  for 
decisionmaking  but  has  les»  _imc  for  making  decisions.  There  is  an 
urgent  requirement  to  gather  large  amounts  of  tactical  information 
quickly,  analyze  the  inf ormation ,  and  disseminate  it  to  field  commanders 
for  decisions  and  actions  necessary  for  defense  against  rapidly  devel¬ 
oping  threats.  As  this  pace  increases,  manual  methods  become  less  ef¬ 
fective  for  the  necessary  q  licker  reaction  times. 

The  Army  is  aware  of  the  problem  and  has  been  seeking  to  improve 
the  tactical  command  and  control  system  by  using  computer  technology 
in  the  decision  process.  This  approach  is  called  the  Tactical  Opera¬ 
tions  System  (TOS)  concept,  and  it  has  involved  an  evolutionary,  itera¬ 
tive  process  of  interactive  studies,  teats,  and  sof t war c/hardware  de¬ 
velopment  efforts.  The  test  discussed  here  was  designed  as  an  operational 
test  of  the  TOS  concept  applied  at  the  division  level. 


Test  Purpose  and  Scope 

The  purpose  of  the  test  was  to  provide  a  data  base  with  which  to 
evaluate  the  TOS  concept  of  automated  data  processing  (ADP)  assistance 
to  the  division  staff,  with  emphasis  on  testing  division  staff's  capa¬ 
bility  to  use  information  input  from  selected  volumes  of  intelligence 
and  combat  information  from  corps.  Test  results  were  to  be  used  in  de¬ 
veloping  requirements  and  reconsnendations  for  a  TOS  at  division  and 
corps. 

The  main  part  of  the  test  was  conducted  as  a  series  of  command 
post  exercises,  usinq  both  the  manual  system  and  a  computer  system-- 
the  TOS  Operable  Segment  (TOS^) ,  a  test-bed  system  designed  to  test 
the  TOS  concept.  Participating  command  and  staff  echelons  were  se¬ 
lected  elements  of  division,  brigade,  and  battalion  staffs.  Control 
organizations  simulated  corps,  direct  support  artillery  battalions, 
and  other  division  units.  The  test  scenario  placed  the  player  division 
in  reserve  for  approximately  4fl  hours  and  in  contact  with  an  enemy  force 
for  approximately  72  hours.  Scenario  events  included  a  mix  of  tacti¬ 
cal  operations,  command  post  displacements,  continuity  of  operation 
conditions,  cownand  post  displacements,  continuity  of  operation  con¬ 
ditions,  and  electronic  warfare  and  security  threats. 


1 


TOS~'  System  Description 

The  TOS*  is  a  limited  application  of  automatic  data  processing  of 
the  tactical  operations  of  a  division  in  the  field.  It  is  a  complete 
computer  system  consisting  of  an  integrated  assembly  of  ADP  hardware, 
software,  personnel,  and  communicat ions  designed  to  support  G2/GJ  op¬ 
erations  within  division  headquarters.  Tactical  information  is  entered 
into  and  retrieved  from  a  central  computer  through  message  input/output 
devices  (MIOD)  using  variable- length,  preformatted  messages.  Informa¬ 
tion  from  the  system  may  be  recorded  in  hard  copy  on  the  MIOD  electronic 
line  printer,  displayed  on  the  display  editor,  or  portrayed  graphically 
on  the  group  display  device  (GDD) . 

The  TOS^  computer  provides  general  services  that  include  editing 
and  validation,  data  access  control,  file  maintenance,  report  genera¬ 
tion,  data  base  query,  and  automatic  dissenu nation  of  information  via 
Standing  Requests  for  Information  (SRI).  In  addition,  there  are  specific 
processes  associated  with  the  three  main  functions  automated  in  the  TOS2. 

Enemy  Situation  (ENSIT).  This  function  stores,  retrieves,  sunsna- 
rlzea,  and  disseminates  selected  portions  of  the  information  utilized 
by  intelligence  elements  in  tactical  operations.  The  categories  of 
information  include  essential  information  about  specific  enemy  units, 
their  combat  effectiveness,  and  history,  plus  spot  report  information 
concerning  enemy  activities,  potential  targets,  losses,  and  other  enemy 
situation  data.  Provisions  are  made  for  an  intelligence  working  file 
and  for  the  preparation  and  dissemination  of  an  intelligence  summary, 
as  required. 

Friendly  Unit  Information  (FRENSIT) .  This  function  provides  for 
the  collecting,  filing,  display,  and  dissemination  of  the  current  status 
of  friendly  units  assigned  or  attached  to  the  division.  The  categories 
of  information  include  location  and  tactical  disposition  of  units;  as¬ 
signment  of  units  within  a  task  organization  or  task  force;  essential 
personnel,  equipment,  and  supply  status  of  units  and  task  forces;  and 
operations  reports.  FRENSIT  contains  the  capability  to  prepare  the 
division  operations  journal  and  situation  reports. 

FRENS I T/ENS I T  (COMMON) .  The  COMMON  function  supports  both  the 
FRENSIT  and  ENSIT  functional  areas  by  providing  the  capability  to  desig¬ 
nate  and  identify  Named  Area  of  Interest  (NAI)  in  the  data  base  and 
monitoring  Standing  Requests  for  Information,  when  used  with  appropri¬ 
ate  ENSIT  or  FRENSIT  messages,  the  COMMON  function  enables  the  division 
staff  to  accomplish,  to  a  limited  degree,  tomplatinq  processes. 
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TUS1*  ADP  Individual  Staff  User  Training  Progi  am 

Prior  to  testing ,  an  ADP  training  program  was  conducted  for  di¬ 
vision  personnel  who  would  operate  the  TOS*  system;  they  were  designated 
alternately  as  staff  users,  trainees,  operators,  or  players.  The  pro¬ 
gram  was  designed  to  train  staff  users  to  employ  the  TOS^  hardware  and 
software  applications  that  they  would  use  in  their  duty  assignments 
during  the  test.  The  program  was  developed  and  implemented  by  training 
personnel  from  Army  Tactical  Data  Systems  Command  Systems  Field  Office 
(ARTADS-CSFO)  and  United  States  Army  Combined  Arms  Center  (CACDA) .  The 
course  consisted  of  individual  as  well  as  unit  training  phases,  but 
training  evaluation  was  concerned  with  the  individual  training  phase. 

Individual  ADP  training  was  designed  to  provide  selected  player 
personnel  with  a  general  knowledge  of  the  capabilities  and  character¬ 
istics  of  individual  TOS*  messages  and  functions  and  the  basic  knowledge 
and  skills  required  to  perform  their  staff  duties  in  the  test.  The 
course  of  instruction  was  divided  into  FRENSIT  and  ENSIT  training  class¬ 
es,  in  accordance  with  the  staff  functions  automated  In  the  TOS^.  The 
programs  of  instruction  for  the  two  classes  were  similar  except  that 
specific  FRENSIT  and  ENSIT  subjects  were  taught  only  in  each  respective 
class.  Players  wore  assigned  to  one  of  the  training  classes  on  the 
basis  of  the  primary  staff  function  they  would  perform  in  the  test. 

Those  with  G3  functions  were  assigned  to  the  FRENSIT  class,  and  those 
with  G2  functions  attended  the  ENSIT  class.  The  program  of  instruction 
is  outlined  in  Table  1. 


Training  Evaluation  Objectives 

One  test  objective  concerned  training  evaluation;  the  test  was  to 
obtain  data  that  permitted  assessment  of  the  training  requirements  neces¬ 
sary  for  the  division  to  employ  TOS 2 .  of  primary  interest  was  evaluation 
of  the  effectiveness  of  the  TOS^  ADP  Individual  Staff  User  Training  Pro¬ 
gram.  One  aim  was  to  determine  (a)  whether  or  not  this  instruction  was 
adequate  for  preparing  the  trainees  to  meet  the  program  objectives  and 
(b)  how  effectively  the  program  had  been  carried  out.  The  other  aim  of 
the  evaluation  was  to  ascertain  if  the  trainees  had  accomplished  the 
program  objectives  at  the  end  of  training.  A  related  interest  was 
identification  of  individual  aptitudes  or  characteristics  that  could  be 
used  as  variables  for  predicting  success  in  TOS*  ADP  training.  The 
variables  could  be  used  as  criteria  for  selecting  trainees  for  assign¬ 
ment  to  ADP  training.  The  present  evaluation  was  undertaken  to  investi¬ 
gate  these  problems. 
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Table  l 


TQS 
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ADI  Individual  Staff 


User  Training  Program  of 


Instruction 


Title 

ENSIT 

class 

S2/G2 

(hours) 

FRENSIT 

class 

S3/G3 

(hours) 

Introduction  to  Staff  User  Course  (TOS2) 

4 

4 

TOS2  Demonstration 

1 

1 

MIOD  Operating  Procedures 

2 

2 

Assembly/Disassembly  of  Cabling 

4* 

4a 

Digital  Data  Terminal 

2* 

2* 

operator  Maintenance/Trouble  Shooting 

2® 

2* 

Communications  Methods 

4* 

4* 

ADP  Tactical  Applications 

4 

4 

Introduction  to  TOS*  Messages 

6 

6 

F-xami nation  (Introduction  to  TOS2  Messages) 

4 

4 

FNSIT  Files,  Formats,  Applications,  and  PE 

24 

0 

FRKNS IT  Files,  Formats,  Applications,  and  PE 

0 

34 

Examination  (FRENSIT/ENSIT,  Files) 

4 

4 

Templating 

2 

2 

Continuity  of  Operations  (CONOPS) 

2 

2 

Discussion  of  Brigade  Relay 

2 

2 

File  Managers 

2 

2 

Staff  Interface  in  TOS2 

6 

6 

Applications  Review 

2 

2 

Graphic  Application 

eb 

0 

Functional  Applications 

18 

18 

GDD  Operations  and  Maintenance 

8b 

0 

File  Manager 

2C 

2C 

Total  Hours 

123 

105 

*Only  Selected  MIOD  Operators  received  these  classes. 

b 

Intelligence  personnel  of  AfcP  only  received  this  instruction. 


Given  only  to  personnel  identified  as  file  managers. 
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METHOD 


Tuinco* 


In  all,  103  officer  and  enlisted  personnel  front  the  Ftrat  Cavalry 
Division  and  a  military  intelligence  battalion  froet  III  Corpa  were  as¬ 
signed  aa  players  in  the  teat  and  aa  trainees  In  the  ADP  training  pro¬ 
gram.  The  backgrounds  and  abilities  in  the  heterogeneous  group  varied 
widely.  Formal  education  ranged  from  9  to  17  years.  Military  grade 
ranged  from  E2  to  05,  and  time  in  service  ranged  from  a  few  months  to 
over  JO  years.  Almost  all  the  officers  had  occupational  specialties 
in  either  combat  arms  or  military  intelligence,  and  large  proportions 
of  enlisted  soldiers  had  either  a  combat  arms  military  occupational 
specialty  (MUS)  or  an  Intelligence  analyst  MOS. 

The  training  classes  were  integrated  with  all  grades,  and  all 
trainees  in  each  class  received  the  same  instruction.  The  FRENSIT 
class  consisted  of  24  officers  and  27  enlisted  personnel,  totaling 
51  trainees;  the  ENSIT  class  consisted  of  15  officers  and  37  enlisted 
personnel,  totaling  52  trainees.  This  trainee  mix  In  both  classes, 
therefore,  meant  that  there  were  wide  variations  in  learning  caj>abllity. 


Questionnaires 

Trainees  were  administered  questionnaires  at  thtee  points  in  the 
program:  at  the  end  of  lessons  covering  maior  subjects,  at  the  end  of 

the  training  program,  and  after  the  test  had  been  completed.  The  end- 
of-lesson  quest lonnal res  were  oriented  toward  obtaining  trainee  opinion 
concerning  the  quality  of  training  and  how  well  the  lessons  had  been 
taught.  The  end-of-traintnq  and  test  questionnaires  were  oriented 
toward  obtaining  trainees'  estimates  of  their  level  of  training  profi¬ 
ciency  before  and  after  the  test. 

FJid-of-besson  Questionnaires.  These  quest lonna l re*  were  designed 
to  obtain  information  concerning  the  effectiveness  of  the  main  compo¬ 
nents  of  lessons  covering  the  major  subjects  in  the  program  of  Instruc¬ 
tion.  F.ach  questionnaire  contained  four  or  five  questions.  Question  1 
asked  trainees  to  rate  the  instruction  in  accomplishing  the  lesson  ob¬ 
jectives  on  a  5-point  scale  from  Very  Effective  to  Very  Ineffective. 
Question  2  called  for  a  yes-no  response  to  whether  they  had  received 
enough  instruction  to  fully  understand  the  lesson  objectives.  Question 
3,  included  for  lessons  having  practical  exercise  sessions,  called  for 
a  yes-no  response  to  whether  they  were  given  enough  hands-on  practice 
with  the  equijwent  to  learn  the  operator  skills  covered  in  the  lesson 
objectives.  Question  4  asked  trainees  whether  they  had  accomplished 
all  the  lesaon  objectives  and  to  indicate  their  Judqmont  with  a  yea-no 
response.  Question  5  presented  the  amount  of  training  time  spent  on 
the  subject  and  asked  trainees  to  Judge  whether  aor^  the  same,  or 
less  time  was  required  to  learn  fully  the  material  "red  by  the 


lesson  objectives.  Following  the  questions,  space  was  provided  for 
recordmq  remark*  and  observat ions .  Table  2  lists  the  trainlnq  sub* 

<•  .  covered  by  end-of- lesson  questionnaires!  these  subjects  accounted 

for  approximately  70%  of  the  trainlnq  t law  in  the  program. 


Table  2 

Trainlnq  Subjects  That  Trainees  Evaluated  with 
End-of-Leason  Questionnaires 


Subject 


Trainlnq 

hours 


MI00  Operating  Procedures  2 
As ses&ly /Disassembly  of  Cablinq  4 
Diqital  Data  Terminal  2 
Operator  Ha intenance /Troubleshoot lnq  2 
CosHunication  Hethods  4 
Introduction  to  TOS'  Nossaqes  6 
ENSIT  Files,  Formats,  Applications,  and  PE  24 
FREHSIT  Files,  Formats,  Applications,  and  PE  34 
Functional  Applications  IS 
File  Manaqer  2 

Total  98 


End -of -Training  and  Test  Quest lonnalres .  These  que s 1 1 onna Ires 
were  designed  primarily  to  obtain  trainee  appraisals  of  their  profi¬ 
ciency  at  the  end  of  training  and  testing,  and  the  relative  amount  of 
training  time  required  to  attain  proficiency  in  each  subject  in  the 
program  of  instruction.  Question  1  consisted  of  two  parts.  In  the 
first  part,  trainees  rated  their  proficiency  on  each  trainlnq  subject, 
using  a  3-point  scale  of  Adequate,  Borderline,  and  Inadequate.  In 
the  second  part  of  Question  1,  trainees  made  a  relative  judgment  of 
the  amount  of  training  time  required  to  attain  adequate  proficiency 
on  each  subject,  when  considering  the  time  allotted  to  each  subject 
In  the  training  program:  they  used  a  3-point  scale  of  More,  Same,  and 
less  Time.  Question  2  asked  respondents  to  rate  how  effectively  the 
individual  and  unit  training  prepared  them  to  perform  their  duties  in 
the  test,  using  a  S-polnt  scale  that  ranged  from  Very  Effectively  to 
Very  Ineffectively.  Question  3  requested  the  trainees  to  give  their 
overall  opinion  of  the  trainlnq  program,  using  a  5-point  rating  scale 
from  Very  Good  to  Very  Poor.  Question  4  asked  for  recommendations  to 
improve  the  training  program. 
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Evaluator  Report  a 

Tha  evaluators  completed  end-of- lesson  reporta  of  the  lessons  for 
all  training  subjects  in  the  program.  Supplemental y  reports  contained 
recorded  observations  of  trainee  performance  in  the  final  exanination 
and  opinions  concerning  the  overall  training  programs. 

The  evaluator  end-of-lesaon  reports  were  designed  to  obtain  evalu¬ 
ator  judgments  of  the  effectiveness  of  training  provided  for  each  sub¬ 
ject  in  the  program  of  instruction.  Item  1  reguired  the  evaluator  to 
rate  the  effectiveness  of  the  training  in  terms  of  four  components: 
lecture  session,  practical  exercise  session,  instructor's  teaching 
ability,  and  total  lesson.  Item  2  called  for  a  yes-no  response  as  to 
whether  or  not  all  lesson  objectives  were  accomplished.  Item  3  asked 
for  the  recording  of  all  noteworthy  observations  and  opinions. 


Final  Examination 


A  final  examination  for  the  training  course  was  given  at  the  con¬ 
clusion  of  ENSIT  and  FRENSIT  training.  The  examination  consisted  of  a 
written  part  and  a  performance  part.  The  written  part  was  made  up  of 
a  series  of  true-false  and  fill-in  questions  concerning  basic  knowledge 
of  system  functions  and  the  preparat ion  of  TQS 2  messages.  The  items 
required  an  understanding  of  the  characteristics  of  message  formats, 
such  as  their  uses,  coding  methods  required  for  entering  information 
c°rr*ctly  into  message  data  sections,  and  system  processes  activated 
by  message  ccsiponents. 

The  performance  part  of  the  examination  was  designed  as  a  test  of 
the  primary  objective  of  the  training  program.  A  set  of  performance 
standards  was  defined  as  the  training  criteria.  The  main  criterion  was 
that  a  trainee  should  be  able  to  send  a  selected  group  of  10  messages 
representative  of  his  section's  ADp  activities  to  the  centrsl  processing 
unit  within  indicated  time  limits.  All  messages  had  to  be  accepted 
(without  error)  by  the  computer  system.  The  test  consisted  of  a  aeries 
of  10  or  11  information  requirements  of  different  types  that  the  trainee 
had  to  translate  into  the  appropriate  TOS*  message  format  and  correctly 
enter  into  the  computer.  Each  trainee  was  seated  at  a  MIOO  by  himself 
and  given  approximately  1  hour  and  20  minutes  to  complete  the  task. 


.'X»sx>jraphlc  Data  Survey 

Demographic  data  were  collected  on  the  enlisted  trainees  in  the 
ENSIT  and  FRENSIT  classes:  24  biographical  measures  and  aptitude  scores 
were  collected  from  the  trainees  and  recorded  from  their  personnel  files. 
Table  3  lists  the  data  elements  collected. 
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Table  J 


Biographical  and  Aptitude  Measures  Included 
in  the  DMoqraphlc  Data  Survey 


A.  Biographical  Scores  and  Measures 


1 .  Grade 

2 .  Sax 

3.  Age 

4.  Civilian  Education 

5.  Tiiae  In  Service 

6.  Military  Education 

7.  Primary  MQS  Score 

8.  Tine  in  Primary  MOS 

9 .  NOS  Group 

10.  School  Trained  NOS 

11.  Typewriting  Experience 

12.  Teletypewriting  Experience 
11.  Keypunching  Experience 

14.  Class  Attendance  in  TGS*  ADP  Training 


l.  Aptitude  Area  Scores  from  the  Army  Classification  Battery 


15. 

CO 

(Combat) 

16. 

FA 

(Field  Artillery) 

17. 

EL 

(Electronics  Repair) 

18. 

GM 

(General  Maintenance) 

19. 

MM 

(Mechanical  Maintenance) 

20. 

CL 

(Clerical) 

21. 

GT 

(General  Technical) 

22. 

ST 

(Skilled  Technical) 

21. 

SC 

(Surveillance  and  Oxiwuni cat  ions) 

24. 

r 

(Operations  and  Food) 

STATISTICAL  ANALYSIS 

The  questionnaire  data  were  analyzed  with  statistical  tests  of  the 
goodness-of-f it  type  that  tested  the  hypothesis  that  the  sasirles  of 
trainee  responses  were  based  upon  experiencing  an  effective  and  efficient 
training  program.  A  set  of  specified  distributions  was  calculated  for 
the  items  in  the  questionnaires  to  represent  those  that  would  be  expected 
frees  a  population  who  had  received  effective  training  in  a  well-run 
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program.  The  null  hypothesis  was  that  there  was  no  difference  between 
the  observed  trainee  response  distributions  and  the  distributions  speci¬ 
fied  for  an  effective  training  program.  Statistical  testa  were  aude  of 
the  data  to  determine  if  there  were  significant  differences  between  the 
observed  proportions  and  the  proportions  e*j>ected  on  the  basis  of  an 
effective  training  program.  When  the  actual  training  program  yielded 
results  that  were  not  as  good  as  the  criterion  specified  for  an  effec¬ 
tive  program,  and  where  these  differences  were  statistically  signifi¬ 
cant,  It  was  assumed  that  the  actual  training  was  not  effective.  The 
level  of  significance  (alpha)  selected  for  the  tests  was  .05. 

T'wo  statistical  tests  were  chosen  for  analyzing  the  questionnaire 
data.  The  chi-square  one-sample  test  (X^)  was  used  for  analyzing  data 
from  items  requiring  yes-no  responses,  which  produced  two  categories 
of  data.  The  Kolmoqorov-Smi rnov  one-sample  test  (D)  was  used  for  ana¬ 
lyzing  data  from  items  calling  for  one  of  several  choices  falling  along 
a  continuous  measurement  scale. 

A  multiple  regression  analysis  was  run  on  the  demographic  data 
collected  on  the  enlisted  operator  trainees  to  discover  variables  for 
predicting  success  in  TQS*  ADF  training.  Dixon's^  ecssputer  program 
for  stepwise  regression  analysis  (R)  was  used  for  making  all  calculations. 


STAFF  USER  TRAINING  EVALUATION 

The  results  are  presented  in  four  sections:  staff  user  training, 
operator  staff  user  training,  commander  and  principal  staff  training, 
and  analysis  of  demographic  variables.  The  first  section  presents 
findings  from  the  end-of- lesson  ratings  and  evaluations;  findings  from 
ratings  and  evaluations  made  at  the  end  of  training  and  testlngj  and 
class  attendance  records  and  results  from  the  final  examination. 


Er.d-of-Leason  Ratings  and  Evaluatl on  a 

Trainee  Ratings.  The  staff  user  trainees  were  divided  into  two 
classes — an  ENS1T  class  and  a  FRENSIT  class.  Each  class  had  a  course 
of  instruction  consisting  of  10  subjects  or  lessons.  The  lessons  for 
both  classes  were  Identical  except  for  the  central  lessons,  which  cov¬ 
ered  ENSIT  flies  and  applications  for  the  ENS1T  class  and  FRENSIT  files 
and  applications  for  the  FRENSIT  class.  The  trainees  rated  5  of  the 
10  lessons  at  the  end  of  each  lesson,  which  accounted  for  approximately 
60  hours,  or  79*,  of  the  training.  The  five  lessons  not  rated  covered 
approximately  16  hours,  or  21»,  of  the  training.  Table  4  summarizes 


lDixon,  w.  J.  Biomedical  Computer  Programs.  University  of  California 
Tress,  1975. 
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the  result*  from  the  end -of- lesson  questionnaires,  listlnq  the  lesson 
titles  and  the  instructional  elements  rated  by  the  trainees. 


Table  4 


Trainee  End-of -Lesson  Ratings  of  Staff  User  Lessons 


Instructional  elements** 


Lesson  rated* 

Traininq 

hours 

Classroom 

traininq 

Hands-on 

training 

Attaining 

lesson 

objective 

Lesson 

time 

MIOD  operating  proc 

2 

S 

S 

S 

S 

Intro  TOS ^  messages 

6 

X 

X 

X 

X 

ENSIT  applications 

34 

S 

s 

s 

X 

FRENSIT  applications 

34 

s 

X 

X 

X 

Functional  applications 

16 

X 

X 

X 

s 

File  manager 

2 

•• 

•• 

•• 

"'lessons  not  rated  i  ADP  doctrine  (4  hr):  COHOPS  (4  hr)  :  staff  Interface 
in  TOS*  (4  hr);  graphic  applications  (2  hr):  and  GOO  ops  and  Mint  (2  hr). 

b 

S  ■  Satisfactory  smdian  ratinq. 

Q 

X  -  Unsatisfactory  setdian  ratinq. 


Trainee  Opinions.  Table  5  presents  trainee  cusi nts  on  the  end-of- 
lesson  questionnaires  about  the  two  main  subjects  of  instruction--ENSIT 
and  FRENSIT  files.  Of  these  convents,  86%  pertained  to  sosm  aspect  of 
the  hands-on  traininq,  and  57%  of  these  (494  of  the  total)  pertained 
to  the  insufficient  time  that  was  devoted  to  hands-on  traininq  and  prac¬ 
tical  exercises. 

Evaluator  Ratings.  All  lesson  elements  received  a  median  ratinq 
of  Effective  for  classroasi  instruction,  instructor's  ability,  and  the 
overall  lesson.  The  practical  exercise  sessions  were  rated  as  border¬ 
line  on  all  lessons,  even  though  the  lesson  objectives  were  judged  to 
have  been  accoeq>lishod. 

Evaluator  Observations.  Generally,  evaluator  conssenta  covered 
several  problems  and  conditions  that  may  have  affected  the  traininq 
pr oqrttn.  Conditions  observed  to  have  a  negative  effect  on  traininq 
were  as  follcws. 
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Table  5 


Trainee  Comments  on  ENSIT  and  FRENSIT  File  Cl 


V 

C 

\ 

u 

Number 

t lacs  made 

Percent 

distribution 

Hoie  hands-on  training  with  the  MIOO's  and 
more  practical  exercises  (PE) 

38 

49 

PE's  inadequate  in  content  and  amount;  they 
should  be  embedded  in  scenario  problems 
so  trainees  can  experience  software  inter¬ 
action  with  message  inputs  and  outputs 

13 

17 

Less  lecturing  and  classroom  instruction 

9 

12 

More  varied  training  on  the  MIOO's 

4 

5 

Too  many  students  assigned  to  MIOO's  during 
hands-on  training 

2 

3 

ADP  training  not  necessary  for  evaluators 

3 

4 

Structure  training  more  toward  specific 
positions 

2 

3 

More  emphasis  on  format  instruction 

2 

3 

More  training  on  interaction  of  files, 
messages,  and  queries 

1 

1 

Allot  more  time  for  teaching  FRENSIT  subjects 

1 

1 

Replacements  should  be  provided  for  students 
who  drop  out  of  training 

1 

1 

Students  should  be  on  special  duty  status 
and  released  from  all  unit  duties 

1 

1 

Total 

77 

100 

1*  A  high  trainee- t o-MIOD  ratio  (between  3  to  1  and  4  to  1)  j 

2.  A  lack  of  enough  assistant  instructors  during  the  early  part 
of  training; 

3.  A  great  range  in  the  backgrounds  and  abilities  of  the  trainees; 

4.  The  doubtful  effectiveness  of  using  half-day  training  sessions 
(full-day  sessions  should  be  used) i 


5.  The  relatively  low  effectiveness  of  the  combination  of  lecture 
anil  hands-on  training  that  was  used  (a  better  balance  between 
lecture  and  hands-on  training  is  needed) ;  and 

»• 

6.  Inadequate  support  fresn  the  trainees'  unit  commanders. 

A  condition  observed  as  having  a  positive  effect  was  the  profi¬ 
ciency  of  the  faculty,  whose  training  and  instruction  quality  through¬ 
out  the  training  program  wore  judged  to  be  very  high  in  all  respects. 


Trainee  Knd-of -Training  and  End-of-Teat  Ratings,  Eva  lust  ions,  and 
Pur  f  ormance  Measures 

Trainee  Ratings.  Table  6  summarizes  the  results  from  the  end-of- 
training  and  end-of-test  questionnaires.  The  table  lists  the  main  les¬ 
sons  comprising  the  staff  user  training  program. 


Table  6 

Trainee  Ratings  on  Training  Proficiency  and  Lesson  Time 


End  of  training* 

End  of 

test* 

Lesson 

Training  Lesson 

proficiency  time 

Training 

proficiency 

Lesson 

time 

MIOO  operating  procedures 

X 

X 

s 

s 

A OP  doctrine 

S 

s 

s 

s 

COMOPS  procedures 

X 

s 

s 

s 

Intro  to  TOS^  messages 

s 

s 

s 

s 

ENSIT  applications 

s 

X 

s 

s 

FRENS1T  applications 

s 

X 

s 

s 

Staff  interface  in  TOS^ 

X 

X 

X 

s 

Graphic  applications 

X 

X 

X 

s 

functional  applications 

s 

s 

X 

s 

GOD  ops  and  maint 

X 

X 

X 

s 

Pile  manager 

X 

s 

X 

s 

ADP  practical  experience 

—b 

X 

X 

*X  ■  Unsatisfactory  median  rating  (less  than  adequate). 

S  •  Satisfactory  median  rating. 

‘  Mo  ratings  made  on  end-of-tralnlng  questionnaire  as  It  was  administered 
subsequent  to  class  training  but  prior  to  practical  exercise. 
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Oi  both  the  end-of-training  and  the  end-of-teat  questionnaire,  the 
trainees  were  asked  to  rate  the  effectiveness  of  individual  and  unit 
training  In  preparing  them  to  perform  their  duties  in  the  TOS2  field 
test.  On  both  questionnaires,  the  great  majority  rated  the  individual 
training  as  effective.  On  the  end-of-test  questionnaire,  a  significant 
proportion  (D  »  2.43,  p  <  .01)  of  trainees,  40%,  rated  their  prepara¬ 
tion  in  unit  training  as  less  than  effective. 

The  trainees  wore  asked  to  give  their  overall  opinion  of  the  ADP 
training.  On  the  end-of-training  questionnaire,  the  majority  rated  it 
as  good.  On  the  end-of-test  questionnaire,  however,  a  significant 
proportion  (D  *  .169,  p  <  .05),  33%,  rated  the  ADP  training  as  less 

than  good. 


Trainee  Recommendations.  Trainees  were  asked  to  make  i  i  i  imwiiiiihIii 
tions  for  improving  the  training  program.  Their  comments  are  given  in 
Table  7  for  the  end-of-training  questionnaire  and  in  Table  8  for  the 
end-of-test  questionnaire.  In  general,  approximately  half  the  train¬ 
ees'  recocwendat ions ,  47%  and  52%,  were  directed  toward  expanding  or 
improving  hands-on  training,  and  another  large  proportion  dealt  with 
other  aspects  of  the  traininq.  Overall,  76%  of  the  recosmtendations  on 
the  end-of-traming  and  63%  on  the  end-of-test  questionnaires  addressed 
the  improvement  of  the  training  methodology.  Another  substantial  pro¬ 
portion  of  the  recommendat ions ,  20%  and  30%  on  the  two  questionnaires, 
concerned  the  overall  course  of  instruction,  pointed  out  weaknesses  in 
the  course  content,  and  suggested  changes  for  improvement. 

Evaluator  Ratings  and  Recommendations.  The  evaluators  rated  the 
effectiveness  of  the  overall  training  program  by  observing  class  and 
test  sessions  and  recording  impressions  of  these  proceedings.  Their 
observations  constitute  a  general  assessment  of  program  effectiveness 
and  level  of  student  proficiency  at  the  end  of  the  training. 

In  general,  the  evaluators  stated  that  students  showed  a  lack  of 
familiarity  with  applying  the  methods  and  procedures  for  processing 
test  problems,  seemed  somewhat  disorganized  in  their  efforts,  and  were 
slow  and  hesitant  in  feeling  their  way  toward  solutions.  With  few  ex¬ 
ceptions,  the  students  usod  the  maximum  time  for  tests  and  even  then 
did  not  complete  all  the  problems.  This  result  indicates  a  need  for 
more  practice  in  performance  testing  to  develop  the  ability  to  detect 
and  correct  input  errors.  Evaluators  also  stated  that  students  need 
exposure  to  test  conditions  during  training,  to  develop  the  self- 
reliance  and  problem-solving  skills  and  abilities  required  for  ADP 
operations . 

Class  Attendance.  Table  9  presents  statistics  of  trainee  daily 
classroom  attendance  for  the  two  ADP  training  classes.  Average  class¬ 
room  attendance  for  the  G2  (ENSIT)  and  G3  (FRENSIT)  classes  was  90% 
and  83%,  respectively.  Of  all  trainees,  91%  in  the  ENSIT  class  and 
81%  in  the  FRENSIT  attended  class  at  least  81%  of  the  time. 
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Table  7 

Trainee  Recommendation*  at  Knd-of-Traininq 


Recoi 


L'tldat  Ion 


Number  Percent 

made  distribution 


More  hands-on,  on-line  training  with  the  HIOD's 

tn  practical  exercises.  32 

Maintain  a  lower  student  to  MIOD  ratio,  two-to-one 

maximum.  8 

Increase  practical  exercises,  minimize  lectures.  6 

feature  more  difficult  problems  to  be  solved 

in  practical  exercises.  6 

More  assistant  Instructors  to  help  students 

during  practical  exercises.  4 

Have  smaller  classes.  6 

Computer  system  breakdowns  Interfered  with 
training;  a  backup  computer  would  minimize 
these  interruptions.  6 

Lectures  were  loo  long.  5 

Training  day  should  be  8  hours  rather  than 

4  hours.  4 

Problem  solving  In  message  fonsats  classes  should 
be  emphasized  by  making  student  select  the 


correct  format  on  the  basis  of  how,  when,  and  why.  4 


A  5-week ,  half-day  course  Is  not  adequate  to 

train  staff  users;  more  training  time.  3 

Design  training  to  be  more  self-paced,  allow  fast 

and  slow  learners  to  progress  at  their  own  pace.  3 

Should  be  more  written  and  performance  tests.  3 

Should  be  more  training  on  message  formats  and 

more  practice  on  writing  formats.  6 

Topics  are  too  general  and  not  oriented  to  the 
specific  functions  of  the  position,  section 
leaders  need  more  training  on  managing  system 
problems  and  functions.  5 

More  training  on  the  Interpretation,  inter¬ 
relations,  and  associations  of  various 
messages  and  queries,  4 

Creator  emphasis  on  SOP’s  and  OFM's  since  they 

regulate  ADP  play.  3 

Staff  should  be  trained  to  function  together.  2 

User's  manual  and  SOP  should  be  better  Integrated 

and  made  more  explanatory.  2 

Need  better  SOP  for  CM&D,  A4P,  and  ATSE  on  the 

subject  of  collection  manages*nt.  1 

The  SOTAS  needs  a  supplement.  1 

Students  should  be  assigned  full  time  to  ADP 

training  (released  from  company  duties).  4 

Students  should  be  assigned  to  TOS  positions 

ensssensurate  with  their  MOS .  1 


27 

7 

5 

5 

3 

3 

5 

4 

3 

3 

3 

3 

3 

J 


4 

3 

2 

2 

2 

1 

1 

3 

1 


TOTAL 


119 


100 


Tdt  ln  8 


Trainee  KecunmvndtUoni  at  End-of-Test 


Recummendat  ton 


Number 

made 


More  hands-on,  on-line  training  with  MlOD's, 

using  drills  and  practical  exercises  (PE's).  30 

Expand  scope  of  PE's  to  Include  operations  tests 
of  all  sections,  a  complete  KTX,  a  brigade  TOC 
eaphas  lz  lug  staff  and  coaasand  Interrelationships, 
and  a  CPX  with  division,  brigade,  and  battalion.  6 

Maintain  a  low  student  to  M10D  ratio  (?-to-l).  5 

Improve  quality  of  PE’s  and  increase  number  of 

Scenario  problems.  4 

Devote  less  time  to  lectures.  4 

Improve  system  reliability  to  reduce  Inter¬ 
ruptions  of  training.  2 

More  format  training  with  the  MIOD's.  1 

In  lectures,  add  demonstrations  (examples)  of 

Individual  functions.  1 

Lectures  were  boring.  1 

More  performance  tests  with  specific  performance 

objectives.  1 

More  class  time;  training  time  is  Inadequate.  6 

(live  more  training  on  following  topics: 

Applying  TOS'  principals  and  using  system 

capabilities.  4 

Interaction  of  files  and  processing  functions.  1 

System  operational  concepts.  1 

Communication  procedures,  querying,  and 

purging  fllea,  and  on  CONOPS.  1 

Interaction  between  messages  files  ESD,  EUS , 

and  CUB.  1 

KRENSIT  formats  DCS,  CSR,  and  RPV  reported  on 

LTOA  formats.  1 

Communications  wiring  and  troubleshooting.  1 

KRENSIT  subjects.  1 

Pile  manager  subjects.  1 

Division  rapid  reaction  force  (DRRF).  1 

battalion  use  of  I'STC  and  ITTDS  messages  In 

changing  altuatlona.  1 

Need  better  understanding  by  staff  supervlaors 

of  ADP  functions.  1 

Generals  and  colonels  need  more  training.  1 

Need  separate  training  programs  for  MIOD 

operators  and  staff.  1 

MIOD  operators  need  training  to  higher  level 

of  competence  and  ability.  1 

Students  should  he  assigned  full  time  to  ADP 

trslning  (released  from  company  duties).  3 

Establish  selection  criteria  for  personnel  for 
ADP  training;  essential  they  have  operations 
experience.  3 


Percent 

distribution 


34 


7 

6 

5 

5 

2 

1 

1 

1 

1 

7 


5 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

X 

1 

1 

1 

3 


3 


TOTAL 


87 


100 


Table  9 


Trainee  Classroom  Attendance  Records 


Attendance 

record 

ENSIT 

class 

FRENSIT  class 

(percent) 

Frequency 

Percent 

Frequency  Percent 

91-100 

36 

59 

27 

50 

81-90 

19 

32 

17 

31 

71-80 

4 

7 

4 

7 

61-70 

0 

0 

1 

2 

51-60 

1 

2 

0 

0 

41-50 

0 

0 

1 

2 

31-40 

0 

0 

0 

0 

21-30 

0 

0 

1 

2 

11-20 

0 

0 

1 

2 

0-10 

0 

0 

2 

4 

Total 

60 

— 

54 

— 

Final  F-xaminat ion .  Descriptive  statistics  of  the  cumulative  fre¬ 
quency  distribution  of  trainee  scores  on  the  written  and  performance 
parts  of  the  final  examination  are  presented  in  Table  10.  Statistics 
on  the  cumulative  frequency  distribution  of  the  number  of  messages  pro¬ 
cessed  correctly  on  the  final  examination  are  presented  in  Table  11. 

The  mean  number  and  mean  percentage  of  messages  processed  correctly 
were  8.29  (83%)  and  7.21  (66%)  for  the  ENSIT  and  FRENSIT  classes,  re¬ 
spectively.  Of  the  trainees,  60%  of  the  ENSIT  class  and  49%  of  the 
FRENSIT  class  made  nine  or  more  correct  message  entries,  which  was  the 
assumed  system  performance  criteria  required.  Tests  for  significance 
of  a  proportion  were  made  to  determine  if  class  performance  satisfied 
the  specification,  and  results  showed  that  the  percentage  of  students 
meeting  the  standard  was  significantly  less  than  the  hypothetical  per¬ 
centage  of  84%  for  both  the  ENSIT  and  the  FRENSIT  classes  (e  proportion 
•  4.57,  p  <  .001  for  ENSIT  and  e  proportion  ■  5.96,  p  <  .001  foi 
FRENSIT) . 


Discussion 

The  primary  purpose  of  the  training  evaluation  was  to  determine 
trainee  proficiency  at  the  completion  of  instruction.  The  training 
objective  was  to  bring  the  students  to  a  level  acceptable  for  perfor¬ 
mance  within  system  specifications  during  the  testing  phases  of  FM  222. 
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Thus,  training  objectives  were  used  a*  the  criteria  for  evaluating 
trainee  achievement  and  the  relative  effectiveness  of  the  instructional 
program. 


Table  10 

Trainee  Scores  on  Final  Examination 


Raw 

score  Frequency 


91-100  38 

81-90  16 

71-80  3 

61-70  2 

51-60  0 

0-50  0 

Total  59 

Mean  91.2 


91-100  21 

81-90  21 

71-80  6 

61-70  1 

51-60  0 

41-50  0 

31-40  0 

21-30  0 

11-20  0 

0-10  0 

Total  49 

Mean  89.1 


Written 


Performance 


Cumulative  Cumulative 

Percent  percent  Frequency  Percent  percent 


ENSIT 


64 

100 

27 

59 

100 

27 

36 

11 

24 

41 

5 

9 

4 

9 

17 

4 

4 

3 

6 

8 

0 

0 

1 

2 

2 

0 

0 

0 

0 

0 

-- 

-- 

46 

— 

90.3 

— 

— 

FRENSIT 

43 

100 

18 

46 

100 

43 

57 

1 

3 

54 

12 

14 

2 

5 

51 

2 

2 

2 

5 

46 

0 

0 

4 

10 

41 

0 

0 

5 

13 

31 

0 

0 

5 

13 

18 

0 

0 

1 

3 

5 

0 

0 

0 

0 

2 

0 

0 

1 

2 

2 

-- 

— 

39 

— 

70.8 

•- 

Note.  Standard  deviation  for  ENSIT  was  7.10  and  10.13  and  for  FRENSIT 
was  8.37  and  28.45. 


Table  11 


Messages 

Processed 

Correct ly 

on  Final  Examinations 

ENSIT 

FRENSIT 

Number 

correct 

Frequency 

Percent 

Cumulative 

percent 

Frequency  Percent 

Cumulative 

percent 

11 

— 

— 

— 

8 

21 

100 

10 

11 

27 

100 

9 

23 

79 

9 

13 

33 

73 

2 

5 

56 

8 

5 

12 

40 

0 

0 

51 

7 

5 

12 

28 

3 

8 

51 

6 

5 

12 

16 

1 

2 

43 

S 

1 

2 

4 

3 

8 

41 

4 

0 

0 

2 

8 

21 

33 

3 

1 

2 

2 

2 

5 

12 

2 

0 

0 

0 

2 

5 

7 

1 

0 

0 

0 

1 

2 

2 

0 

0 

0 

0 

0 

0 

0 

Total 

41® 

100 

— 

39 

100 

— 

Mote.  Standard  deviation  for  ENSIT  was  1.68  and  for  FRENSIT  was  3.32. 

®Does  not  Include  scores  from  five  students  whose  message  processing 
performance  was  obstructed  by  computer  malfunction. 


Trainee  Evaluation.  In  the  staff  user  training  program,  each  staff 
user  was  required  to  perform  an  information-processing  task  by  sending 
10  basic  types  of  messages  to  the  ccsnputer  within  specified  time  limits. 
All  messages  had  to  be  accepted  by  the  computer,  thereby  indicating  that 
the  entry  of  all  information  elements  was  essentially  correct.  The 
evaluation  criterion,  therefore,  was  whether  or  not  each  trainee  cor¬ 
rectly  entered  10  messages.  The  percentage  of  trainees  who  met  this 
criterion  was  the  empirical  measure  used  to  estimate  the  degree  to  which 
the  training  objectives  were  achieved. 

If  the  criterion  of  10  correct  message  entries  is  used,  and  other 
factors  are  not  taken  into  consideration,  then  35%  of  all  the  trainees 
achieved  this  standard.  Examination  of  the  distribution  of  scores  for 
the  ENSIT  class  (Table  11),  however,  reveals  that  one-third  of  that 
class  made  9  correct  message  entries.  The  distribution  of  scores  for 
the  class  is  positively  skewed  and  dichotomized  between  those  making 
9  and  10  correct  entries  and  those  making  8  or  fewer.  The  10-message 


18 


criterion  Joes  not  conform  to  the  dichotomy  end  does  not  reflect  the 
natural  division  in  performance  differences  brought  about  by  the  test. 
The  distribution  implies  a  doubt  as  to  whether  there  are  any  real  dif¬ 
ferences  in  proficiency  between  trainees  making  a  score  of  9  or  10  on 
the  test,  whereas  there  is  more  confidence  that  scores  of  8  and  9  miqht 
indicate  meaningful  differences.  A  reluctance  to  exclude  the  trainees 
who  had  scores  of  9  from  consideration  stemmed  from  the  fear  of  maklnq 
an  error  in  judgment  that  miqht  result  from  an  appraisal  that  was  in 
conflict  with  the  dichotaaxius  distribution  of  scores. 

A  criterion  chanqe  to  nine  or  more  correct  messaqe  entries  expands 
the  ranqe  to  include  all  scores  that  could  possibly  meet  the  trainmq 
objectives.  When  this  criterion  is  applied,  the  results  show  that  S4* 
of  the  trainees  achieved  the  standard.  This  flqure  is  believed  to  be 
a  more  accurate  indicator  of  the  extent  to  which  the  traininq  objectives 
were  achlevod. 

Program  Evaluation.  The  assessment  of  proqram  effectiveness  in¬ 
volves  consider inq  the  organirat lonal  qoals  that  the  proqram  was  de- 
siqned  to  accomplish.  If  100*  criterion  achievement  was  the  desired 
orqanirational  qoal,  then  the  proqram  failed  to  attain  its  objective 
by  a  wide  margin.  On  the  other  hand,  when  the  proqram  desiqn  is  studied 
it  becomes  obvious  that  the  conditions  necessary  for  total  success  were 
not  established.  The  primary  deficiency  was  the  absence  of  any  pretrain 
inq  selection  process  to  screen  out  individuals  who  lacked  the  potential 
to  succeed.  All  individuals  assigned  to  the  proqram  were  accepted  into 
it.  A  few  of  the  trainees  had  no  more  than  1  or  4  months  total  time  in 
service,  and  others  had  little  understanding  of  G2  or  G3  operations. 
These  trainees  did  not  have  the  necessary  background  knowledge  for 
achieving  the  traininq  objectives.  The  wide  diversity  in  the  trainees' 
experience,  background,  and  ability  was  apparent  to  instructors  and 
evaluators  alike. 

This  situation  carries  the  implication  that  the  potential  for 
achieving  traininq  objectives  was  a  normally  distributed  variable  among 
the  trainees.  This  implication,  in  turn,  tends  to  modify  the  goal  of 
the  training  proqram.  If  it  is  assumed  that  trainee  potential  was  nor¬ 
mally  distributed,  and  that  the  short-term,  fast-paced  tralnir.q  proqram 
was  effective  for  traininq  only  trainees  whose  potential  was  at  least 
high  as  one  standard  deviation  below  the  mean,  then  the  maximum 
proportion  of  trainees  that  could  be  expected  to  achieve  the  traininq 
objectives  would  be  about  86*.  Considering  the  wide  diversity  in  train¬ 
ee  ability,  this  level  seems  to  be  a  more  realistic  goal.  Assessing 
the  program  with  this  value  increases  the  effectiveness  rating  to  64*» 
this  latter  flqure  appears  to  provide  the  most  equitable  estimate  of 
the  effectiveness  of  the  traininq  proqram.  The  assessment  Indicates 
that  the  proqram  was  fairly  effective,  but  room  for  improvement  remains. 
The  problem  of  findlnq  ways  to  improve  proqram  effectiveness  will  be 
taken  up  next  by  evaluating  the  strengths  and  weaknesses  of  the  proqram. 


Moth  trainees  and  evaluators  gave  consistently  high  ratings  to 
the  quality  of  the  program  of  instruction  and  the  teaching  ability  of 
the  instructors  on  most  of  the  lessons.  The  course  content  was  well 
locumented,  in  the  form  of  a  highly  organized,  extensively  detailed 
staff  user  training  manual  supported  with  supplementary  volumes  of 
standing  operating  procedures  (SOP) .  Trainees  used  the  trainlnq  man¬ 
ual  continuously  as  a  classroom  text  during  lectures  and  as  an  opera¬ 
tors*  guide  and  reference  during  hands-on  training.  The  subject  matter 
was  comprehensive  and  covered  all  aspects  of  the  basic  system  operations 
with  which  staff  users  were  involved.  Most  technical  matters  and  staff 
user  functions  seemed  to  have  been  thoroughly  covered.  As  an  introduc¬ 
tory  course,  the  program  of  instruction  was  satisfactory  in  covering 
the  basic  functions.  The  only  cosgilaints  concerned  methods  of  perform¬ 
ing  in  certain  staff  positions.  The  lectures  were  well  organized, 
clearly  presented,  and  comprehensible.  The  instructors  had  a  good  grasp 
of  the  subject  suit  ter  and  were  able  to  provide  satisfactory  explanations 
for  most  of  the  questions  raised  by  the  trainees.  During  the  lectures, 
instructors  used  slide  projectors  extensively  as  visual  aids,  which 
helped  to  clarify  many  procedural  details.  The  lecture  phase  of  train¬ 
ing  was  used  effectively  for  teaching  the  large  volume  of  knowledge 
comprising  the  course  content. 

The  trainee  and  evaluator  ratings  and  comments  clearly  Identified 
three  major  factors  that  costoined  to  substantially  diminish  the  train¬ 
ing  capability  inherent  in  this  program.  The  most  Influential  factor 
was  that  the  trainee  load  was  too  heavy  for  the  limited  training  resour¬ 
ces  made  available.  The  program  had  the  capacity  Co  train  perhaps  70 
to  80  trainees  under  th«  training  conditions  prevailing  In  FM  222  >  how¬ 
ever,  approximately  190  staff  users.  Including  controllers,  were  admitted 
to  trainlnq.  For  FM  222,  the  classes  were  simply  enlarged  to  double 
their  capacity,  and  the  original  training  schedule  was  followed  without 
mod 1 f icat ion • 

The  second  factor  was  that  the  aaiount  of  tisw>  scheduled  and  the 
training  technique  used  for  the  practical  exercises  were  Inadequate. 

The  common  complaint  of  trainees  was  that  they  did  not  experience 
enough  hands-on  training  and  could  not  apply  the  knowledge  learned 
from  lectures  by  actually  performing  data  processing  operations  with 
the  computerized  system.  MIOD  time  allotted  for  one  user  had  to  be 
divided  up  among  three  or  four  users.  Thus,  some  trainees  actually 
experienced  only  one-fourth  the  amount  of  hands-on  training  they  should 
have  had.  The  practical  exercises  were  designed  for  individual  learn¬ 
ing  wherein  a  trainee  could  develop  the  important  skills  necessary  for 
independent  interaction  with  the  computer  system.  In  many  sessions, 
the  opportunity  was  not  there,  because  the  practice  became  merged  into 
a  group  effort.  Three  or  four  trainees  would  gather  around  one  MI00, 
and  they  all  would  participate  in  working  through  a  problem  that  one 
of  them  should  have  been  doing  independently.  The  effects  of  this  lack 
of  practice  surfaced  dramatically  in  the  performance  exam,  where  many 
trainees  experienced  great  difficulty  in  structuring  the  problems  and 
working  them  out  in  a  logical  or  systematic  manner. 
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Another  week  aspect  of  the  practical  exercises  identified  by  the 
trainees  was  that  there  was  not  enough  direct  interaction  with  the  TC6 
software  system  in  the  training.  Much  of  the  training  was  devoted  to 
the  fundamental  problems  of  message  format  selection,  structuring  the 

ihto  the  format  fields,  and  data-entry  and  error-correction  opera¬ 
tions.  The  trainees  obtained  little  experience  in  how  the  TOS  software 
system  processed  the  input  data,  updated  files,  and  sent  the  output 
back  to  the  originator  in  a  feedback  loop,  simulating  how  the  actual 
system  would  function. 

.ralnees  also  acquired  little  experience  in  observing  how  their 
inputs  affected  various  files  and  functions  already  in  the  system's 
meswry,  or  how  the  new  information  was  used  with  the  old  information 
under  a  specified  set  of  conditions.  The  practical  exercises  did  not 
include  any  practice  tests  similar  to  the  final  performance  examination; 
therefore,  until  the  final  performance  examination,  they  never  faced 
hands-on  test  situations  involving  difficult  problems  that  would  expose 
their  inadequacies  of  knowledge  and  skill.  The  perfors^nce  examination 
was  their  first  experience  with  operating  the  system  without  assistance 
from  others.  A  performance  examination  scheduled  to  be  given  on  the 
first  examination  during  the  second  week  of  tralnlnq  was  cancelled  be¬ 
cause  of  computer  breakdown;  thus  the  trainees  were  deprived  of  valuable 
information  about  their  ability  for  lnde(>*ndent  operation. 

The  third  factor  was  that  significant  percentages  of  trainees  felt 
that  too  little  time  was  allotted  to  teaching  the  main  lessons  on  the 
ENSIT  and  FRENSIT  files.  These  lessons  covered  a  large  asiount  of  sub¬ 
ject  matter  in  a  relatively  short  time.  The  pace  of  instruction  wss 
fast,  and  emphasis  was  placed  on  the  fundamentals  of  each  topic.  Less 
complete  coverage  was  given  to  the  higher- level  and  store  intricate 
application  of  functions  during  tacticsl  operations.  Trainees  indi¬ 
cated  that  they  desired  store  training  time  for  learning  the  main  sub¬ 
jects  and  subjects  dealing  with  functional  applications  and  information 
management . 


OPERATOR  STAFF  USER  TRAINING  EVALUATION 

These  findings  present  results  from  measures  of  operator  staff 
user  training  taken  at  different  times  in  the  test  program.  Ratings 
and  evaluations  were  made  at  the  end  of  each  lesson,  at  the  end  of  the 
training  program,  and  again  at  the  end  of  the  T06^  field  test. 


End-of -Lesson  Ratings  and  Evaluations 

The  training  program  for  operator  staff  users  wss  a  subordinate 
part  of  the  staff  user  program  of  Instruction.  It  consisted  of  four 
lessons  taught  to  a  selected  group  of  approximately  57  enlisted  staff 
users,  with  the  purpose  of  teaching  operators  the  skills  involved  in 
moving  and  setting  up  the  MIOD  for  operation. 


Trainee  Ratings.  Table  12  presents  a  suwnary  of  the  end-of- lesson 
questionnaire  results.  On  the  items  dealing  with  the  effectiveness  of 
instruction,  the  trainee  median  rating  of  the  quality  of  instruction 
was  effective  or  very  effective  on  all  four  lessons.  On  the  items  con¬ 
cerned  with  the  practical  exercise  part  of  the  lessons,  the  trainees 
felt  they  had  received  adequate  practice  in  two  of  the  lessons,  and  in¬ 
adequate  practice  on  the  other  two.  Forty-three  percent  of  the  train¬ 
ees  indicated  they  had  insufficient  practice  on  the  digital  data  termi¬ 
nal.  On  the  lesson  on  communication  methods,  30%  Indicated  they  had 
insufficient  practice  on  the  equipment.  Most  of  the  trainees  felt  that 
they  had  accomplished  the  lesson  objectives  on  all  four  lessons  and 
that  the  time  allotted  for  each  lesson  was  adequate. 


Table  12 

End-of -Lesson  Rating  of  Instruction 


_ _ Instructional  elements _ 

Attaining 

Practice  lesson  Lesson 

Lesson  Instruction  session  Objectives  time 


Assembly  and  disassembly 
of  cabling 

Digital  data  terminal 
Operator  maintenance  and 
troubleshooting 
Communication  methods 


A 

A 


AAA 
X  A  A 


AAA 
A  X  A 


A 

A 


Note .  A  indicates  adequate;  X  indicates  inadequate. 


Trainee  Opinions.  Table  13  presonts  a  classif ication  and  tabula¬ 
tion  of  trainee  remarks  in  the  end-of-lesson  questionnaire  covering  the 
four  lessons.  The  most  frequent  remark  made  was  that  there  were  too 
many  trainees  for  the  number  of  MIOO's  available,  and  the  third  most 
frequent  was  that  they  needed  more  practical  exercise  time  on  the  MIOD. 
These  two  directly  related  problems  accounted  for  52%  of  all  the  comments. 

Evaluator  Ratings.  A  summary  of  the  median  ratings  of  the  lesson 
elements  for  the  four  lessons  for  selected  MIOD  operators  is  shown  in 
Table  14. 
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Table  13 


Trainee  (.'amenta  on  MIOO  Operator  Lessons 


Remark 

Number 

made 

Percent 
dlstr Ibut ion 

Too  many  trainees  per  MIOO;  class  too  large 
for  amount  of  resources  available 

14 

33 

Distractions  in  classroom  degraded  learning 

conditions 

9 

21 

Need  more  practical  exercise  on  the  equipment 

8 

19 

Items  of  equipment  nonoperat lonal  or  not 

aval lable 

4 

9 

Need  more  detailed  instruction  on  specific 

topics 

3 

7 

Assistant  instructors  not  available  to 

trainees  needing  guidance 

3 

7 

Instructor  was  a  good  teacher 

3 

4 

Total 

43 

100 

Table  14 

Evaluator  Ratings  of  MIOO  Operator  Lessons 

Lesson  element 

Median  rating 

Lecture  session 

Effective 

Practical  exercise  session 

Borderline 

Instructor's  ability 

Effective 

Total  lesson 

Effective 

Lesson  objective  accomplished 

Yes 

Evaluator  Observations .  In  general,  the  evaluatora  noted  the  same 
problems  identified  by  the  trainees — too  many  trainees  for  the  nvsaber 
of  MIOD's  available  and  Insufficient  practical  exercise  time  on  the 
MIOD's.  There  were  18  HIOO's  available  and  60  trainees,  for  a  student- 
to- MIOO  ratio  of  3.3  to  1.  The  class  instructor,  who  had  worked  with 
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the  TOS  system  for  several  years  and  was  familiar  with  the  technical 
aspects  of  TOS  training,  held  the  definite  opinion  that  the  ideal  train- 
ee-to-MIOO  ratio  is  1  to  1  and  should  not  be  higher  than  2  to  1 .  In 
addressing  the  practical  exercise  time,  the  evaluators  stated  that, 
with  three  or  four  trainees  assigned  to  each  MIOD,  it  was  difficult 
for  everyone  to  obtain  an  adequate  amount  of  time  for  practice. 

End-of-Training  and  End-of-Toat  Ratings.  In  general,  the  end-of- 
training  and  end-of-tcst  questionnaires  covered  the  same  four  lessons 
as  listed  in  Table  12.  At  the  end  of  the  training,  40*  of  the  trainees 
felt  that  there  should  have  been  more  time  devoted  to  training  on  as¬ 
sembly  and  disassembly  of  cabling.  At  the  end  of  the  test,  53%  felt 
that  more  time  should  have  been  given  to  training  on  operator  main¬ 
tenance  and  troubleshooting. 


Discussion 

The  purpose  of  this  subordinate  evaluation  was  to  appraise  the 
adequacy  of  the  operator  staff  user  training  in  preparing  the  trainees 
to  perform  within  Bystem  specifications  during  the  testing  phases  of 
FM  222.  The  evaluation  indicated  that  the  overall  objective  of  the 
training  was  attained;  however,  the  lack  of  sufficient  time  and  equip¬ 
ment  adversely  affected  training  quality. 


COMMANDER  AND  PRINCIPAL  STAFF  TRAINING  EVALUATION 

On  the  items  dealing  with  effectiveness  of  instruction,  the 
trainee  median  rating  was  Effective  and  the  evaluators  rated  the  lec¬ 
ture  session,  instructor's  ability,  and  the  total  lesson  as  Effective. 
The  majority  of  trainees  stated  that  the  time  devoted  to  each  lesson 
was  adequate  for  learning;  however,  the  evaluators  rated  the  practical 
exercise  session  as  borderline.  The  lesson  objectives  for  all  lessons 
were  judged  to  have  been  accomplished. 

Although  this  evaluation  indicated  that  the  lesson  objectives  were 
attained  and  that  the  training  of  the  commanders  and  principal  staff 
was  satisfactory  overall,  the  problem  of  insufficient  practical  exer¬ 
cise,  as  noted  in  the  other  training  phases,  was  also  noted  in  this 
phase  of  training. 


ANALYSIS  OF  DEMOGRAPHIC  VARIABLES 

Data  from  all  the  enlisted  trainees  in  the  ENSIT  and  FRENSIT 
classes  who  completed  the  final  examination  were  analysed  in  a  step¬ 
wise  regression  analysis.  The  sample  site  was  48  subjects.  Table  3 
lists  the  24  biographical  measures  and  aptitude  scores  Included  in  the 
analysis  as  independent  variables.  The  dependent  variable  was  the 


sum  of  the  normalized  scores  from  the  written  and  performance  portions 
of  the  final  examination. 


Hesulta 

Table  15  presents  the  results  of  the  stepwise  regression  analysis. 
Of  the  24  independent  variables,  4  were  found  to  have  significant,  posi¬ 
tive  correlations  with  the  examination  score.  The  ccsnbination  of  all 
four  variables  accounted  for  a  significant  proportion  of  the  total 
variance  in  the  examination  scores  (F  ■  11.72,  df  ■  4/42,  p  <  .001). 

The  value  of  the  coefficient  of  determination  R2  equaled  .5275,  which 
means  that  approximately  53*  of  the  variance  of  the  examination  scores 
was  accounted  for  by  the  four  attribute  variables  in  combination — the 
CT.  primary  MOS ,  CO,  and  SC. 


Table  15 

Summary  of  Stepwise  Regression  Analysis  for  Data  from 
MIOD  Operators  in  ENSIT  and  FRENSIT  Classes 


Variable  entered 

df 

F  to  enter 

Increase  in  R* 

GT  (general  technical) 

1/42 

21.15 

.3198 

Primary  MOS  score 

1/42 

8.26 

.1075 

CO  (combat) 

1/42 

4.44 

.0536 

SC  (surveil lance/comm) 

1/42 

4.15 

.0466 

Value  of  R2 

•• 

•• 

.5275 

The  last  column  in  Table  15  gives  the  proportions  of  variance  due 
to  the  individual  variables.  These  results  show  that  when  the  four 
variables  are  taken  together  the  GT  score  is  the  most  powerful  pre¬ 
dictor  and  accounts  for  about  32*  of  the  variance.  The  primary  MOS 
score  accounts  for  about  11*  of  the  variance,  and  the  combat  (CO)  and 
surveillance  and  communications  (SC)  scores  account  for  about  5*  each 
of  the  variance.  It  appears  that  all  four  attribute  variables  are 
meaningful  predictors  of  MIOD  operator  training  proficiency.  Since 
the  effort  and  expense  are  virtually  equivalent  for  obtaining  any  one 
of  these  scores,  or  all  four,  it  would  be  advantageous  to  use  them  all 
to  maximize  predictive  power. 
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Discussion 


This  analysis  suffers  frcan  one  major  weakness:  The  sample  size 
may  not  have  been  large  enough  to  insure  high  reliability  of  the  re¬ 
sults.  (The  method  of  multiple  regression  analysis  is  known  for  its 
low  reliability,  and  that  problem  is  minimized  by  using  large  samples.) 
Ker linger  and  Pedhazur^  noted  that  some  authors  recommend  that  the 
sample  size  be  at  least  30  subjects  per  attribute  variable.  In  this 
analysis,  the  advice  calls  for  a  sample  size  of  120  rather  than  the  48 
that  was  used.  Since  our  sample  may  not  have  been  sufficient,  the  re¬ 
sults  should  be  viewed  as  preliminary  findings  that  have  uncovered  four 
attribute  variables  that  show  promise  as  predictors  of  aptitude  for 
TOS  operator  staff  user  training. 


^Ker linger ,  F.  N. ,  fc  Pedhazur,  E.  J.  Multiple  Regression  in  Behavioral 
Research.  New  York:  Holt,  Rinehart,  and  Winston,  1973. 
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